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V»© (57) Abstract: A method is described for measuring system latency and latency jitter on a near real-time basis regarding packets of 
information moving from sources (figure 1, item 46) to destinations (item 46). The method for measuring system latency and latency 

^ jitter includes time-stamping packets of information (figure 2A, item 13) as they pass through a plurality of locations between the 
source and destination of these packets, with the time-stamping being performed by real-time clocks set to a universal standard (figure 

O 3). The time-stamped packets are analyzed to determine latency and latency jitter. The measurements of the system latecy and jitter 

^ can be performed based upon packet types so as to obtain accurate information concerning different types of business bandwidths 

^ associated with an observed network. 
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REAL TIME MESH MEASUREMENT SYSTEM STREAM LATENCY 
AND JITTER MEASUREMENTS 

CROSS REFERENCE TO RELATED APPLICATIONS 

The present application discloses subject matter which is disclosed and may be claimed in the 
following international applications as identified by applicant's attorney's docket nos. 
402-127.2-1, 402-127.4-1, 402-127.5-1 and 402-127.8-1. 

Application No. 402-127.2-1 is directed to a closed loop method for baselining business 
bandwidth in a network environment 

Application No. 402-127.4-1 is directed to analysis of business bandwidth for control of same. 
Application No. 402-127.5-1 is directed to the application of closed loop control to control 
of business bandwidth. 

Application No. 402-127.8-1 is an extension of 402-127.2-1. 402-127.3-1, 401-127.4-1 and 
402-127.5-1 with respect to exportation of information in a multiple management environment 
(multiple users with different SLAs). 

TECHNICAL FIELD 

The present invention is directed to an implementation/method that identifies how to create 
a new measuring "sense" for measuring business bandwidth performance. This new "sense" 
is the ability to "see" (observe & measure) in real time business bandwidth latencies and the 
rate of change of the latency, known as latency jitter. 

BACKGROUND OF THE INVENTION 

In a networked environment, the length of time for packets to travel from a source to a 
destination can vary for many reasons, including delays due to routers, communication, media, 
and the like. Although attempts have been made to determine latency and latency jitter 
through use of remote monitoring devices (RMONSs) and the like, such efforts have generally 
been unsatisfactory. It is desirable to make such measures in an accurate and precise manner. 
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DISCLOSURE OF THE INVENTION 
Thepresent process able^ 

time bandwidth latency and latency jitter. Observations are continuously and simultaneously 
made over the network in order to measure: 
• Multiple end-to-end paths. 

This ability .0 continually and simultaneous!, measure latency (both end-«c-end and for each 
component) ailows the measurement "sense" to idamfy me specific component ma, may be 
causing la*ncy jitter ma, exceeds thresholds and cause in,ermi«en, business bandwKUh 
problems. Additionally, the ability ,0 synchronize measurement taken from mulhple 
.ools/end points (and correlate/analyze in mal nme mis infonnanon) is »ha. is referred fr, as 
the Real Time Mesh Measurement System. 

Definitions: 

Token: a single packet or packet train. It represents an application profile (such as two 
packets, a pause, then three more packets). 



: a packet train simulates the traffic generated by specific applications. 



Packet Train: 



Tracer: the same as "marker". 



Marker: an 
customer traffic) 



inserted packet pair that frames (e.g., beginning and end) a stream of actual 



The present invention defines a new type of network service monitoring and measurement 
system that can fully measure key performance characteristics of specific network serv.ce 
offerings, targeted for identifying specific customer utilization and performance metr.cs 
utilizing those specific services. This new monitoring system/method is primarily based on 
the ability to insert management traffic onto a specific customer service or a specific serv.ce 
provider's service channel. The two types of inserted traffic are: 
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Tokens: tokens simulate different network traffic types. 
Markers/Tracers: these inserted packets frame (one at the beginning and one 
at the end) streams of network traffic for specific services and/or specific 
customers. 

The key components necessary to enable this new monitoring service to utilize these inserted 
packets and fully capture the critical performance statistics are as follows: 

Multiple hardware modules (bandwidth monitoring devices) that can be 
deployed throughout the network (system) that can transmit and receive these 
tokens and tracers. 

A distributed time service that can accurately synchronize, time stamp, and 
correlate the transmit and receive characteristics of the inserted traffic. 
A token building and distribution application that distributes configured/pre- 
defined tokens that will simulate customer/service traffic. 
Monitoring and performance assessment modules that can locally collect and 
aggregate these inserted packets and then analyze them to produce performance 
metrics (latency, jitter, dispersion,m and gap analysis - packets within a token). 
Data collection and analysis modules that collect the raw and analyzed 
simulation data from multiple points and then perform further analysis to create 
additional performance metrics (delivery assurance, delivery reliability, 
effective bandwidth, and available bandwidth). 

BRIEF DESCRIPTION OF THE FIGURES 

Figure 1 is an overall diagrammatic representation of a typical deployment 
environment of the present invention in which various networks are interconnected for 
providing bandwidth necessary for implementing one or more service level agreements. 

Figure 2 is a block diagram of the bandwidth monitoring device interconnect module 
forming part of a bandwidth monitoring device. 

Figure 3 is a block diagram of the master control module of the bandwidth monitoring 

device. 

Figure 4 is a block diagram showing how the bandwidth interconnect module and the 
master control module are put together to form the bandwidth monitoring device. 

Figure 5 is a flow chart showing how new business bandwidth measurements, 
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including latency jitter and end-to-end component measurements ara made according » the 

present invention. m 

Figure6isaflow chart illustrating how the distributed tool time clock x S .mplemented. 

Figure 7 is a flow chart showing how measured transaction latency is measured. 

Figure 8 is a flow chart showing how token latency and token jitter are measured. 

Figure 9 is a high level block diagram of the director console according to the present 

invention. . . - 

Figura 10 is a block diagram of the director console interfaces wnh blocks used for 

determination of latency and latency jitter identified with the lettering U. 

Figure 11 is a block diagram of the director console control interface module. 

Figure 12 is a block diagram of the director console control data flow arid, modu.es 
^iated with detemnnafion of latency and toncy jitter identified with ttte lettering U. 

Figme 13 is a block diagram of the database analysis with modules associated wrth 
latency and latency jitter identified with the lettering U. 

Figure 14 is a Mock diagram of database access associated with lareney and latency 

jitter measurements. 

Figure 15 is an overall block diagram of the director console architecture for serv.ee 
level agreement monitoring controls. 

Figure 16 is a block diagram of the director console appliance interface wrth modules 
associated with latency and latency jitter identified with the lettering U. 

THE BEST MODE FOR CARRYING OUT THE INVENTION 

M bus, seen in Figure 1, a typical deployment of the presen, invention comprises a phnaUty 

edge routers 4a and building routers 44. Edge routers (sw«ch fabric) are typically deployed 
a, the edge of one wide area network (WAN) associated with an interne, service provrder 
(BP, m between a WAN and a local area nettvork (LAN) as shown in Figure 1, The edge 
devices (appliances) (bandwidth monitoring devices a, the edge of different netwotks) are 
positioned so as to monitor bandwidth passing through an edge router typicaUy to a bu.ld.ng 
Ler 44 from whence fire bandwidth is communicated through LANs, ummafcly to one or 
m ore network enabled devices 46 such as PCs, prime* ami the like and/or to server famta 47^ 
Depending upon the level of monitoring, a specie! type of ba^width monitoring device such 
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as a network enabled device bandwidth monitoring device 40 particularly suited for 
connection to network enabled devices 46 can be employed. Such monitoring devices 
typically are positioned between a LAN and a plurality of network enabled devices 46. In 
addition, server farm bandwidth monitoring devices 40 can be positioned between server 
farms 47 and LANs or interbuilding routers 52 as shown in Figure 1 . 

For the methods according to the present invention to be implemented, in addition to the 
various types of bandwidth monitoring devices, the present invention further includes one or 
more director consoles 54 which pool and further analyze the bandwidth data associated with 
the bandwidth monitoring devices. It should be noted that Figure 1 is a typical deployment 
associated with the methodologies according to the present invention. Limitless other 
variations can be contemplated. The underlying concept however is for the bandwidth 
monitoring devices to be associated at locations where communications from one network 
environment to another is desired to be monitored, such as between a WAN of an internet 
service provider and an edge router interfaced to a building router associated with an office 
building and the like. As seen in Figure 4, each bandwidth monitoring device 40 includes 
a bandwidth interconnecting module 41 as shown in Figure 2 and a master control module 43 
as shown in Figure 3. The master control module can communicate with one or more 
bandwidth interconnecting modules via backplane bus 45. 

The present invention provides a process that identifies how to measure business bandwidth 
performance and in particular, to determine business bandwidth latencies and the rate of 
change of such latency known as latency jitter through observations that are continuously 
made in order to determine multiple end-to-end paths for packets traversing from a source to 
a destination, as well as individual components that make up such paths and which may 
therefore affect latency and latency jitter. 

In order to determine such latency and jitter concerning business bandwidth, it is necessary 
that the present invention be implemented through a distribution of bandwidth monitoring 
devices 40, wherein real-time time stamping is performed. Details of the bandwidth 
monitoring device, including the bandwidth interconnecting module 41 and the master control 
module 43 are shown in Figures 2 and 3 respectively. Through use of these multiple physical 
tools the task of interacting with key components in the network for measuring transaction 
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Utoncy from each lamdwidti, monitoring device to each key component is accomplished. 
These measurements are performed by collecting information from me bandwidth monttonng 
devices throng! synchronised time^amped data and performing an analysis on snch data at 
a director console 54. 

Figures 5 - 8 are the overall flow charts which describe dr. implementation of the method of 
determining .atoncy and latency jitter with respect to received bandwidth over a network 
mpotogy snch as that shown in Figure 1. These flow char* nse the hardware shown m 
Figurea 1, 2. 3 and 4 concerning the bandwidm monitoring device 40 and the director console 
54 Detaila concerning the architectore of the director console 54 arc preyed in Frgures 9 - 
16 win, specific modules associated with latency and latency jitter measurements tdentrfled 
by the tetter U". Furthermore, the flow of information with respect to the bandwrdm 
monitoring device 40, including the bandwidm intercomrection module 41 rmd fire maaer 
control module 43, as well as the director console 54 are shown by flow lines contauung 
unbent which serprentially represent the stops for performing the rasks and subrasks relatod 
to latency and latency jitter measurements according to the present mventron. 

Referting again to the flow chart shown in Figures 5 - 8, Figure 5 is particularly directod to 
ft. stops for performing new business bandwidm measurement, including .atoncy. .atoncy 
jitter and end-to-end component measurement, The flow chart shows the steps for 
performing such measurements. The distributee tool time Cock module 70 is further 
described in Figure 6 witt. respec, to the steps for obtaining disttibuted tool fime clock 
measurements. 

Figure 7 is a flow chart with respect to measurement of transaction latency while Figure 8 
is a flow char, of the measuremen. process for token latoncy and jitter. Witt, respect to detatts 
concerning autoteaming and autobasefining, cross-reference is made to applicants co-pendmg 
app.ication emitted "Method for Automatic*.., Basetining Business Bandwidth" (attorney 
docket number 402-127.2-1) filed on even date hereof. 

Overall the present invention has a disttibutod set of bandwidth monitoring devices which 
automatic* identify key component to be measured based upon accurato rea.-t.me 
teaming and autobaselining modules with respec. to key users and applications wmch arc 
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deemed to be important by the user. Such information can be configured by the user. 



The present invention further provides the ability to deploy these distributed tools at key 
locations in the network such as those shown in Figure 1. The ability for the tools to interact 
with each other, coupled with the component measurements allows the overall system to 
generate latency "maps" which are a list of devices, type of latency measurements, the 
accuracy and the rate thereof are described in Figure 5. 

Through this information, the system can configure and autotune the distributed tools for 
specific path measurements which thereby enables critical end-to-end and component views 
to be developed such as those set forth in Figure 8. 

The use of virtual token ring technology on these specified paths allows stream-based 
messages to circle a virtual ring by enabling component to component stream latency and 
jitter measurements. See Figure 8 for additional details concerning such virtual token 
methodology. 

Furthermore, the present invention provides the ability to use and configure different message 
types such as unicast, multicast, COS, POS, RSVP, which thereby allows for measuring 
variations in latency due to message type. 

Time Clock Synchronization 

The present invention, through its distributed tool time clock, is able to accurately synchronize 
these clocks coupled with the above virtual token, and allows for highly accurate latency and 
jitter measurements to be made for specific paths and specific components. Furthermore, the 
present invention provides the ability to continually calibrate and synchronize the distributed 
clocks based upon transaction response time and other observed or monitored data. See 
Figure 6 for additional details concerning the distributed time clock. 

Thus the present invention provides that all latency measurements with respect to the network 
can be accurately time stamped. By so doing, the collection analysis and coirelation of 
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multiple tractions and (virtual token) stream latency measurements are made. The analysrs 
output forms a complex end-to-end/componen, latency and jitter measurement (also known 
as Mesh). Furthermore, foe ability «o store ttme stemps for specifc application messages as 
foey are observed passing through Ore bandwidth monitoring devices, allows for the measonng 
■■sense" .0 automatical* vahdate (and calibrate/tune measurements) foe latency measurement 
results. It should be pointed out that user messages are tagged as tmcers as shown m the 
ggmes. This validation is accomplished by selecting specific message types to he traced and 
collecting/anting tracer message time stamps, and comparing them again* Mesh 
measurements. 

The Mesh measurements are the ability tt. use the Mesh ttacer comparison to modify the Mesh 
token path (rate, specific information) so as .0 more accurately measure latency which allows 
a closed-loop tuning process for the Mesh measurements. 

FWS 5 - 8 represent an operational description of how the measurement components and 
synchronize mefoods interact with each other .0 form mis new business bandwrdttr 
(multiple end-to-end, component to component) latency and latency jitter measurement, 
Further deteils concerning foe flow pafoa shown in Figures 5 - 8 are presented in F.g-re for 
foe flow paths shown therein, Figure 3 for foe flow paths shown Aerein, and Figure 9 w,fo 
respect ro foe director console. Further deteils concerning foe diattibutod time clock and 
.ateney/jitter measurements are presented in Figures 9 - 16 ami a!so in Figures 2A and 3A. 

in Summruy, foe mefoodology according «o foe present invention provides a new business 
bm d„idfo measurement system that measures on a continuous and simultaneous hears, 
multiple eod-to-ond and component latency and latency jitter (also known as Mesh). The 
componentsoffoehusinessbandwidfoMeshsenseinoludefoe ability ,0 automatically idenhfy 
specificpoints in foeMe^end nodes, severs, tools, network product) that are critical pomts 
for observation arch as high usage, important semces, specific funnel points/bottleneck 
nations; foe ability to automatically develop a latency map (lis. and opemtional sfote) for 
each latency measurement, thereby allowing for user input so as .0 rune foe latency map; the 
abiWy to accurately synchronize muHiple larency measurements and ro accurately time-atamp 
foese measurements, thereby enabling accurate remote coUection and date analysis; where.,, 
me new latency/jitter measurements based on both virtual token ring technology and 
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synchronized clocks and time stamps. Such measurements allow stream data types (such as 
voice/video/UDP) to be measured accurately for critical metrics such as latency, latency jitter 
verses "burst load". 

Furthermore, the Mesh sense provide for the ability to collect latency measurements from 
multiple bandwidth measuring devices and to create a real-time Mesh view of latency and 
latency jitter throughout the network. It is also able to validate latency measurements by 
identifying application message types to be used as tracers in the Mesh; by accurately time 
stamping tracer messages passing through bandwidth measuring devices; and by collecting and 
analyzing tracer messages by a centralized validation module such as located in the director 
console. Furthermore, the Mesh sense components provide for the ability to continually 
observe change in latency which would require recommendation changes in the latency map, 
as well as the addition of specific points for latency measurements and the latency rate for 
more accuracy. 

Having described the invention, what is claimed is: 
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1. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations, comprising the steps of: 

1) time-stamping packets of information that pass through a plurality of 
locations between sources and destinations of packets, said time- 
stamping with a real-time clock set to a universal standard; and 

2) analyzing the time-stamped packets to determine latency and latency 
jitter as said packets traverse from sources to destinations. 

2. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 1, wherein a real-time mesh view 
of latency and latency jitter is generated regarding the entire network. 

3. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 2, wherein only packets of 
information of a particular type or types are time-stamped. 

4. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 3, wherein only packets of 
information of a particular type or types are time-stamped. 

5. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 2, wherein tracer packets are 
generated and time-stamped as these tracer packets traverse the network. 

6. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 1, wherein tracer packets are 
generated and time-stamped as these tracer packets traverse the network. 

7. A method of measuring system latency and latency jitter regarding packets of 
information from sources to destinations according to claim 2, wherein the latency map can 
be tuned by a user based upon the latency measurements. 
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1 8. A method of measuring system latency and latency jitter regarding packets of 

2 information from sources to destinations according to claim 1, wherein specific points? in the 

3 network are identified as of higher priority and are locations where packets of information are 

4 time-stamped. 

1 9. A method of measuring system latency and latency jitter regarding packets of 

2 information from sources to destinations according to claim 1, wherein said time-stamping, 

3 time clock is et to a universal time standard. 

1 10. A system for measuring system latency regarding packets of information from sources 

2 to destinations, comprising: 

3 1) means for time-stamping packets of information that pass through a plurality 

4 of locations between sources and destinations of packets, said time-stamping 

5 with a real-time clock set to a universal standard; and 

6 2) means for analyzing the time-stamped packets to determine latency and latency 

7 jitter as said packets traverse from sources to destinations. 
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